Medium streaming distribution system

ABSTRACT

A medium streaming distribution system reduces effects of packet loss in a network before the packet reaches radio base station. A medium distribution device for packet-transmits via the base station, a medium stream to the network by a real time transmission protocol. A packet analyzer monitors the packet arriving at the radio base station and transmits feedback information associated with loss of a packet to the medium distribution device. Based on the feedback from a relay device and a terminal device of the medium stream, the transmission rates from the medium distribution device to the relay device and from the relay device to the terminal device are obtained to provide a greater transmission rate in a surplus band for re-transmission or a forward error correction.

TECHNICAL FIELD

The present invention relates to media streaming, and more particularlyrelates to streaming in a case where delay on a stream transmission pathbecomes larger mid-way through the transmission path, such as the casewhere a media stream is transmitted to a client via a last-hop wirelesscellular link. Note that, though a wireless cellular link is taken as anexample of the transmission path of a large delay in the followingexplanations, the present invention is not limited thereto.

BACKGROUND ART

In recent years, due to striking advances in communication technologies,it has become possible to perform multimedia delivery by means ofwireless communication. In order for a client that receives wirelessdata delivery to playback multimedia in real time, QoS control over acommunication path (transmission rate control and packet loss resiliencecontrol) become important.

Non-Patent Document 1, shown below, discloses that a wireless basestation comprises an RTP monitoring agent after a shaping point (a flowcontrol point). An RTP (Real-time Transfer Protocol) monitoring agentmonitors a multimedia stream that arrives at the wireless base stationand is to be transmitted via a wireless cellular link, and feeds back toa media server the information indicating the congestion status in awired network from the media server to the wireless base station. Theclient that receives data packets wirelessly from the wireless basestation transmits reception acknowledge information to the media serveraccording to RTCP protocol.

The media server determines the congestion status in the wired networkand the link error condition over the wireless link based on theinformation fed back from the RTP monitoring agent and the receptionreport from the client. When a packet loss has occurred due to thecongestion in a wired network, the media server reduces the media streamtransmission rate, and when packet loss has occurred due to an error inthe wireless link, it increases the packet loss resiliency of theencoding of the media. The packet loss resiliency can be increased by,for example, increasing the rate of intra-frame coding in videoencoding.

In this type of conventional technique, once a packet loss occurs,considerable time is required before packet loss resiliency isincreased. In a case where packets containing an I-picture in a videostream is discarded by network queues in response to congestion in thewired network and packets of multiple subsequent P-pictures are thentransmitted via a last-hop wireless link, the client is unable toreconstruct these P-pictures since the I-picture needed for reference toreconstruct the P-pictures is not available. Furthermore, even when aP-picture is discarded, no subsequent P-pictures dependent on thatP-picture can be reconstructed until the next I-picture is received. Ingeneral TCP communications, a packet that is lost is retransmitted basedon feedback information. However, in streaming where the media isreconstructed and played back in real time, retransmission may be toolate in such a case where the buffer size of the client is small. Whenpackets are discarded due to the congestion in a wired network,reconstruction errors occur not only for the discarded packet but alsofor the P-pictures contained in subsequent packets successfullyreceived.

Therefore, there is a need to reduce the error propagation phenomenonwhere a packet loss occurring in the network upstream from the basestation of a low-speed communication such as a wireless communicationhas an effect on reconstruction of the media contained in the subsequentpackets.

DISCLOSURE OF THE INVENTION

A media streaming delivery system according to an aspect of the presentinvention comprises: a media delivery apparatus for transmitting a mediastream in packets to a network according to a real time transferprotocol; and a wireless base station connected to the network fortransmitting the media stream by a low-speed communication such as awireless communication. The system further includes a packet analysisdevice for monitoring the packet arriving at the wireless base station,and transmitting feedback information relating to a packet loss to themedia delivery device.

According to another aspect of the present invention, there is provideda media delivery apparatus included in the above-mentioned system.

According to an embodiment of the present invention, a packet analysisapparatus transmits a sequence number included in a packet header to themedia delivery apparatus as feedback information.

Furthermore, in an embodiment of the present invention, the mediadelivery apparatus determines packet loss based on the feedbackinformation and, depending on the determined packet loss, modifies themedia stream so as to reduce the influence of the packet loss.

According to an embodiment of the present invention, the media stream isa movie including a sequence of I-pictures and P-pictures. The mediadelivery device includes a storage for storing a plurality of mediastreams for one movie including at least a first media stream includingI-pictures in a first arrangement and a second media stream includingI-pictures in a second arrangement which is different from the firstarrangement.

The media delivery apparatus further includes switching means for, inresponse to the determination of the packet loss, for a destinatedlocation where the loss has occurred, selecting a media stream in whichthe first I-picture after the picture in the lost packet appearsearliest among the plurality of the media streams and switching themedia stream to be sent to the selected media stream. This can reducethe error propagation phenomenon where a packet loss has an effect onplay back of the media contained in the subsequent packet.

In an embodiment of the present invention, there are provided mediastreams with different frequencies of intra-frame coded I-pictures, andswitching is made to the media stream with a higher frequency ofI-pictures in response to packet loss.

Furthermore, according to another aspect of the present invention, themedia delivery apparatus further includes an encoding device forgenerating the media stream. The encoding device is arranged to generatea media stream starting with an I-picture in response to thedetermination of packet loss.

According to an embodiment of the present invention, the encoding deviceincreases the frequency of I-pictures at least for the media streamtransmitted to the client where the loss has occurred in response todetection of the packet loss based on the feedback information from thepacket analysis apparatus.

Furthermore, according to an embodiment of the present invention, themedia delivery apparatus receives client feedback information regardingpacket reception from a wireless terminal that is the destination of themedia stream, and determines whether the packet loss has occurred in thewired transmission in the network or in the wireless transmission fromthe wireless base station based on the client feedback information andthe feedback information from the packet analysis apparatus.

Furthermore, in an embodiment of the present invention, the mediadelivery apparatus performs packet retransmission based on the feedbackinformation on the packet loss.

Furthermore, according to an embodiment of the present invention, atransmission rate from the media delivery apparatus to the relayapparatus and a transmission rate from the relay apparatus to theterminal are obtained based on the feedback from both the media streamrelay device and the terminal, and a surplus band is used forretransmission or forward error correction on the side of greatertransmission rate, thereby improving communication quality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing concepts of an embodiment of thepresent invention□.

FIG. 2 is a block diagram showing a construction of an embodiment of thepresent invention.

FIG. 3 is a block diagram showing a construction of an alternativeembodiment of the present invention.

FIG. 4 is a diagram showing examples of video compression codingformats.

FIG. 5 is a diagram showing examples of MPEG I-picture and P-picturesequences.

FIG. 6 is a sequence chart showing a communication sequence inaccordance with a first embodiment and a second embodiment of thepresent invention.

FIG. 7 is a block diagram showing concepts of yet another embodiment ofthe present invention.

FIG. 8 is a block diagram showing detailed constructions of theembodiment in FIG. 7.

FIG. 9 is a sequence chart showing communication sequences in accordancewith a third embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the present invention will be explainedwith reference to the drawings. FIG. 1 is a diagram showing an overallconstruction of an embodiment of the present invention. A media deliverydevice 11 performs real time delivery of multimedia. The multimediaincludes images, audio, text, graphics, and the like, and streams ofeach are packet-transmitted, and are reconstructed and played back inreal time by a receiving device. In the following explanations, thesemedia are described with respect to a video stream. The media deliverydevice 11 can deliver the media stream in the form of on-demand videodelivery, or in the form of a broadcast.

When a media delivery request comes from a wireless terminal 19 such asa mobile phone, a portable terminal (PDA), a computer provided with awireless communication device, the media delivery device 11 packetizesthe media stream, attaches an RTP (Real-time Transport Protocol) headerand an IP header thereto, and transmits this via a wired network 21. Thepackets are routed according to an Internet Protocol (IP), and arrive ata wireless base station 10. The IP is a connectionless communicationprotocol, and when congestion occurs on the network, packets inoverflowing network queues may be discarded from the network.

The wireless base station 10 transmits the packets received from thenetwork 21 to the wireless terminal 19. According to the presentinvention, a packet analysis device 15 is provided to the wireless basestation 10 and monitors the media stream packets that arrive at thewireless base station 10. The sequence of the media stream packets sentto a particular wireless terminal 19 can be identified by means of apair of source and destination addresses included in the IP header and apair of source and destination port numbers included in the UDP header.As will be explained in detail below, the RTP header is included in thepacket, and the RTP header includes a sequence number that is unique tothat packet in the media stream.

When detecting the flow of a sequence of packets having the same sourceaddresses and destination addresses and also the same source portnumbers and the destination port numbers, the packet analysis device 15gives feedback (ACK) of the packet numbers contained in the RTP header,to the media delivery device. The media delivery device 11 can detectpacket loss from this feedback information. When the media deliverydevice 11 detects that an I-picture packet was lost from the packetsequence numbers, the media format is switched to a format with agreater frequency of I-pictures.

With this operation, the wireless terminal 19 can receive the I-picturesearly, and therefore a decline in the quality of reconstruction of thevideo can be prevented.

Referring next to FIG. 2, more detailed explanation is given regardingan embodiment of the present invention. The media delivery device 11stores video contents A, B, C, and D into a multimedia content database27 in two different formats. That is, the video A is stored as videocontents A1 in a first format, and video contents A2 in a second format.Similarly, the video B is stored as contents B1 in the first format andcontents B2 in the second format, and the video C is stored as contentsC1 in the first format and contents C2 in the second format.

FIG. 5 shows examples of the two formats. FIG. 5(A) shows an example ofthe first format, and FIG. 5(B) shows and example of the second format.The second format is set with a higher I-picture frequency than thefirst format. In the diagram, I indicates an intra-frame codedI-picture, and P indicates the P-picture that is prediction-coded basedon the I-picture. When there is another preceding P-picture between theI-picture and the P-picture, the P-picture is prediction-encoded basedon the I-picture and based on the preceding P-picture. In thisembodiment, the movies are encoded in accordance with MPEG 4.

According to the MPEG 4 standard, the frames (pictures) of an inputtedimage is classified into I-pictures, each of which is encoded usinginformation solely on that frame independent of images in the otherframes, and P-pictures, each of which is predicted based on anI-picture, and residual prediction error is encoded for transmission.

The I-picture and the P-picture are each converted by DCT (DiscreteCosine Transform) into DCT coefficients. The DCT coefficients and motionvectors information are entropy coded. At this time, one picture isdivided into 8×8-pixel blocks, and DCT is performed on each block. Fouradjacent blocks are combined into one macroblock (MB), and multiplemacroblocks are combined as one group of blocks (GOB). As shown in FIG.4(B), multiple macroblocks are placed in one packet and transmitted.

In this embodiment, the media delivery device 11 performs the on-demandmultimedia delivery. In the diagram, the video content is only shown,but audio content which will be synchronized with the video andreconstructed is also stored in the database similarly, and is sent inparallel with video content.

A format switching section 29, in response to an instruction from afeedback control section 35, switches the content extracted from thedatabase 27 between the first format and the second format.

A multimedia stream detecting section 37, which is included in thepacket analysis device 15, detects the packet containing the RTP headerfrom among the IP packets arriving at the wireless base station 10, anddetects the multimedia stream based on the pair of the source addressand the destination address, and the pair of the source port number andthe destination port number, of the packet. An RTP header monitorsection 39 monitors the RTP header contained in the multimedia stream IPpackets that are detected by the multimedia stream detecting section 37,and inspects the packet sequence number contained in the RTP header.When loss of a sequence number is detected from inspecting the sequencenumbers across multiple packets, the RTP header monitor section 39transmits information for identifying the lost IP packet to a feedbackgenerator 41. In an embodiment, the information for identifying the IPpacket includes the source address, the destination address, and thelost RTP packet's sequence number. Furthermore, in another embodiment,the RTP monitor section 39 transmits not the lost RTP packet's sequencenumber, but rather the sequence number of an RTP packet each time theRTP packet is detected, to the feedback generator 41.

The feedback generator 41, upon receiving each RTP packet, returns thesequence numbers thereof to the source address as an ACK. In analternative embodiment, the feedback generator 41, in response to theinformation transmitted from the RTP header monitor section 39,determines whether the RTP packet is lost within a predetermined periodbased on the sequence numbers of the sequence of RTP packets, andtransmits the sequence number of the lost RTP packet to the sourceaddress. In the feedback in this case, the RTCP can be extended and sentas well. These are sent to the wired network 21.

The feedback control section 35 of the media delivery device 11 receivesthe feedback information from the packet analysis device 15, and inresponse to reception of the lost packet's sequence number, transmits amedia format switching instruction to the format switching section 29.In another embodiment, the feedback control section 35, based on the ACKinformation that is sent from the packet analysis device 15, monitorslost packet sequence numbers, and when a lost one is detected, transmitsthe media format switching instruction to the format switching section29.

In all the embodiments, the feedback control section 35, when it isdetected that the packet is lost, transmits the format switchinginstruction to the format switching section 29. When the lost packet isfor a P-picture, the influence is less than in the case of theI-picture. Therefore, if the effect of the switching will be relativelysmall (e.g., when there are only a few P-pictures before the nextI-picture, or the like) the format switching may be omitted.

Now, assuming that the media A1 in the first format is provided to thewireless terminal 19, the format switching section 29 switches thestream to the media A2 which is in the second format. Referring back toFIG. 5(A), in the media A1 steam, it is assumed that what is lost iseither one or a plurality of multiple packets containing a macroblock ofI-pictures 51, or a packet containing a macroblock of P-pictures 51 a.The format switching section 29, in response to the instruction from thefeedback control section 35, switches the media to A2 in the secondformat, and, after the I-picture 51 or P-picture 51 a in the media A1 islost, starts the transmission from the I-picture 57 that appears firstin the media A2.

At the wireless terminal 19, since one or more packets of the I-picture51 or packets of the P-picture 51 a are lost, subsequent P-pictures 51b, 51 c, 51 d, and 51 e which are encoded depending upon the I-picture51 or the P-picture 51 a cannot be reconstructed properly. However, themedia delivery device 11 switches the media format to the second formatwhere the I-picture frequency is higher, and starts transmission fromthe I-picture 57 in the second format shown in FIG. 5(B). Therefore, thewireless terminal 19 can properly reconstruct the media from theI-picture 57.

FIG. 6 shows an example of communication and processing sequences in themedia delivery system in accordance with the present invention.

The content that is extracted from the database 27 is packetized at apacketizing section 31, and is then sent out from a stream transmissionsection 33 to the wired network 21. Referring to FIG. 6, the streamtransmission section 33 executes an RTP processing routine 111 and anRTCP processing routine.

The RTP processing routine adds a compression code header foridentifying the format of the compression coding to compressed datacorresponding to one packet, and also adds the RTP header, the UDPheader and the IP header. FIG. 4(A) shows an example of such a packetformat. The RTP header contains the packet sequence number. An initialvalue of this sequence number is determined with a random number, andserial numbers starting from this initial value are added to eachpacket. The RTP header contains an SSRC identifier (SynchronizationSource identifier). For this identifier the same value is added tomultiple streams that should be handled in combination, for example, avoice stream and an image stream of the same user.

The processing routine 111 adds the RTP packet with the RTP header, theUDP header, and the IP header, for transmission according to a UDP/IPprotocol. The IP header includes the source address indicating thetransmission source, and the destination address indicating the client.Furthermore, the UDP header includes the source port number indicatingthe port number of the transmission source, and the destination portnumber indicating the port number of the client.

FIG. 4(B) shows how a picture in one frame is divided into multiplepackets and sent in accordance with the MPEG standard. For example, theI-picture, because of its large information volume, is divided intomultiple packets and then sent. When much congestion occurs on thenetwork, a portion of the packets of the I-picture may be discarded, andthere is a possibility of the complete I-picture not arriving at thewireless terminal 19. In FIG. 4(B), GOB indicates the group of blocks,and MB indicates the macroblock.

The RTCP processing routine 113 exchanges the control information withthe network transmitting the RTP packets. The RTCP (Real-Time TransportControl Protocol, RTP Control Protocol) is a protocol that supplementsthe RTP, where the transmission device and the receiving device notifythe control information to each other, whereby flow control, clocksynchronization, and synchronization between media such as between audiodata and video data are achieved.

In advance of transmission of the RTP-packet, the RTCP processingroutine 113 turns the SDES (Source Description) which is a descriptionabout the information source into an RTCP packet 115 and transmit itonto the network using the UDP/IP protocol with the IP header and theUDP header added. After this, the RTP processing routine transmits RTPpackets 117, 119, and 121 in sequence to the network.

In FIG. 2, a shaping point 13 placed before the wireless base station 10represents one or a plurality of relay devices which govern flowcontrols in the wired network on the path from the media delivery device11 to the wireless base station. The wireless base station 10 isprovided with, for example, a layer 3 switch as a counterpart componentof the shaping point for buffering the packets received from the wirednetwork 21 at high speed and providing them to the wireless section at aspeed adapted for the processing speed of the wireless section of thewireless base station 10.

Referring back to FIG. 6, based on the RTCP packet 115 and thesubsequent RTP packets 117, 119, and 121, the packet analysis device 15of the wireless base station 10 detects that they are parts of the samemedia stream, and calculates a packet loss rate, jitter and the like(131) from the packets received over a predetermined period. It thenputs these pieces of information to an RTCP packet directed to the mediadelivery device 11, and transmits this packet onto the wired network.

Each time the packet analysis device 15 detects an RTP packet from thesame media stream from the media delivery device 11, it transmits areception acknowledge signal (ACK) containing the sequence number ofthat RTP packet to the media delivery device 11. In an alternativeembodiment, the packet analysis device 15 does not return the ACK eachtime the RTP packet is received, but monitors if the sequence numbers ofthe RTP packets belonging to the same media stream are contiguous. Whenthe packet analysis device 15 detects that a sequence number is skipped,it transmits a reception failure signal (NACK) containing the sequencenumber of the lost RTP packet to the media delivery device 11.

In the RTCP processing routine 113 of the media delivery device 11, whenan RTCP packet 133 which is a response to the RTCP packet 115 sent outis received, depending on the packet discard rate and other controlinformation contained therein, the transmission rate of the media streamto be sent out is modified. Furthermore, when the packet loss ratedetected in the RTCP processing routine 113 is high, the feedbackcontrol routine 35 transmits, to the format switching section 29 (FIG.2), a control signal to switch the subsequent media stream format to theformat with the higher I-picture frequency.

In the RTP processing routine 111 of the media delivery device 11, eachtime ACK packets 105, 107, and 109 from the wireless base station 10 arereceived, the sequence numbers, which are contained in the ACK packets,of the RTP packets received from the wireless base station 10, aretransferred to the feedback control section 35. The feedback controlsection 35 inspects whether or not any of the sent packets did notarrive at the wireless base station 10, which is to say whether or notany packets are lost (141). When the feedback control section 35 detectsthat a packet is lost, the feedback control section 35 then judgeswhether or not the data contained in that RTP packet is an I-picture(143). When the lost packet is an I-picture, the above-mentioned formatswitching procedure (145) starts immediately.

At step 143, when it is determined that the lost packet is in aP-picture, it is then determined whether or not the packet is lost up toa reference for retransmission set when a P-picture is lost (147). Whenthe packet of the P-picture is lost, the influence is less than in thecase of the I-picture. Therefore, when a predetermined reference valueis reached the format is switched (149), and until this reference isreached the format is not switched. For example, the reference may beset so that the format is not switched in the case where there are onlya few P-pictures before the next I-picture.

The wireless section of the wireless base station transmits the receivedIP packet by wireless. The wireless terminal 19 receives the wirelesscommunication and reconstructs the media stream. Transmitting andreceiving IP packets by wireless is a publicly known technique, so thatdetailed explanation thereof is omitted.

Depending on the construction of the wireless terminal 19, the wirelessterminal 19 can transmit the ACK signal to the media delivery device 11.Specifically, a wireless terminal that conforms to the 3GPP standardcannot transmit ACK packets as the packet analysis device 15 does. Whenthere is no need to conform with this standard, the wireless terminalcan be constructed to transmit ACK packets. The feedback control section35 compares the feedback information from the packet analysis device 15with the information from the wireless terminal 19, whereby the qualityof the wireless communication from the wireless base station 10 to thewireless terminal 19 can be known. When it is judged that the quality ofthe wireless communication is low, the feedback control section issuesto the format switching section 29 an instruction to switch the mediaformat to the format with the higher frequency of I-pictures. Theformats for the media are not only two types. Many types of formats areavailable, and the format may be switched among many formats dependingon the communication status. For example, when transmissions are beingperformed in a format with a high frequency of I-pictures, ifsufficiently good communication status is detected, then the format canbe switched to the format with the lower frequency of I-pictures. Inthis way, the media delivery device 11 can adjust the load of the streamtransmission.

Referring next to FIG. 3, explanation is given regarding anotherembodiment of the present invention. In FIG. 3, elements that aresimilar to the embodiment in FIG. 2 are indicated with the samereference numerals. At the media delivery device 11, inputted images 30are compressed/encoded in real time by an encoding section 28, and thenturned into packets by the packetizing section 31, and then sent to thewired network 21 as the media stream. The inputted images 30 may beimages that have been recorded in advance and stored in a storagedevice, or may be images that are captured in real time using a videocamera as in a relay broadcast.

The encoding section 28 encodes the inputted images according to theMPEG4 standard which has already been explained. The encoding section 28has a transcoding function, which in response to an instruction from thefeedback control section 35, modifies the frequency of the I-pictures inthe encoding. That is, when, based on the feedback information from thepacket analysis device 15, it is detected that packets including amacroblock of I-pictures are lost in the media stream that arrives atthe wireless base station 10, the encoding section 28 switches theencoding so as to increase the frequency of the I-pictures.

Next, explanation is given regarding another embodiment for performingretransmission of the lost packet, based on the feedback from thewireless base station.

In the wireless link, delay time is normally long. The time for goingback and forth between the media delivery device and the wireless basestation, is much shorter than the time for going back and forth betweenthe wireless base station and the wireless terminal client. Therefore,based on the size of the receiving buffer provided to the wirelessterminal (i.e., the extent to which packet arrival can be delayed due toretransmission of the packets), when any packet is lost on the wirednetwork from the media delivery device to the wireless base station andin the wireless link from the wireless base station to the wirelessterminal, as a form of restoring the lost packet, it is determinedwhether to retransmit the lost packet on the wired network or wirelesslink, or to perform forward error correction (FEC), so that thethroughput of each communication path can be optimized. FEC isspecifically realized by using Reed-Solomon encoding or other errorcorrection encoding commonly known to those skilled in the art.Furthermore, in order to further improve resistance to burst errors,interleaving can be performed, which is also commonly known to thoseskilled in the art. A commonly known example of using interleaving toimprove the ability to correct burst errors is CIRC (Cross InterleavedReed-Solomon Code) used, for example, for CD-DA.

In order to perform packet retransmission and FEC at their proper times,three modes may be provided as a form of restoring the lost packet onthe two communication paths, based on the size of the wirelessterminal's receiving buffer. These modes are explained in detail below.

FIG. 7 shows an overall construction of a media stream delivery systemprovided with such a function. Constituent elements similar to FIGS. 1,2 and 3 are indicated by the same reference numerals. The wireless basestation 101 is provided with a first feedback section 71, an FEC decoder72, a shaping point 13′ that performs flow control at the wireless basestation, an FEC generator 18, a second feedback section 73, and awireless transmission device 14. The FEC decoder 72 functions, in themode for using FEC to restore the lost packet on the network 21 (Mode 3explained below), to restore the lost packet using error correctioncoding information that is sent over on the network 21, and remove errorcorrection code and transmit the main body of the data to the nextstage. The shaping point 13′, when packets reach in excess of the buffercapacity of the wireless base station 10, discards a portion of thepackets. That is, the shaping point 13′ is the means for adjusting therate of packets into the wireless link. The FEC generator 18 functions,in the mode for using the FEC to restore the lost packet on the wirelesslink between the wireless device 14 and the wireless terminal 19 (Mode 2and Mode 3 explained below), to generate the error correction code to beattached to the data stream that will be sent over the wireless link.

FIG. 8 is a diagram showing an embodiment in a case where, in the systemshown in FIG. 7, the media delivery device employs a format switchingsystem. It corresponds to the embodiment in FIG. 2. A multimedia streamdetecting section 37′ detects the packet containing the RTP header, fromamong the IP packets arriving at the wireless base station 10, anddetects multimedia stream based on the pair of the source address anddestination address, and the pair of the source port number and thedestination port number, of the packet.

An RTP header monitor section 39′ monitors the RTP headers contained inthe IP packets of the multimedia stream detected by the multimediastream detecting section 37′, and inspects the sequence numbers of thepackets contained in the RTP headers. Based on the inspection of thesequence numbers across multiple packets, when it is discovered that anysequence number is lost, the RTP header monitor section 39′ transmitsinformation for identifying the lost IP packet, to the feedbackgenerator 41′. In one embodiment, the information for identifying the IPpacket includes the source address, the destination address, and thelost RTP packet's sequence number. Furthermore, in another embodiment,the RTP monitor section 39′ does not transmit the sequence number of thelost RTP packet, but transmits the sequence number of each RTP packetthat is detected to the feedback generator 41′.

The feedback generator 41′ returns the sequence number of the packet tothe source address as an ACK upon receiving each RTP packet. In analternative embodiment, the feedback generator 41′, in response to thetransfer of the information from the RTP header monitor section 39′,judges whether or not there is any lost RTP packet during apredetermined time based on the sequence of RTP packet's sequencenumbers, and then transmits the sequence number of the lost RTP packettoward the source address. In yet another alternative embodiment,instead of making a response to the source address each time theindividual RTP packets are received or each time the lost RTP packet isfound, the response is given with respect to multiple RTP packets at atime. To explain it in more detail, the received/lost data indicatingwhether each packet was received normally or was lost, is returned forevery N number of packets. The number N of packets that are groupedtogether is determined such that the response delay time falls withinsuch a range that enables retransmission of the packet from the sourceaddress in time.

FIG. 9 is a sequence chart showing a communication sequence in thesystem in FIG. 8. As was explained with reference to FIG. 6, the mediadelivery device 11 has the RTP processing section 111 and the RTCPprocessing section 113. The RTCP processing section 113, beforetransmitting a PTCP packet, sends out the RTCP packet 115 containingsender information (SDES) to the wired network 21 toward the wirelessbase station 10. The first feedback section 71 of the wireless basestation 10, in response to the RTCP packet 115 and the subsequent RTPpackets 117, 119, and 121, calculates the packet loss rate and thenetwork transmission delay time of the wired network 21, and then mountsthis information onto the RTCP packet 133 and transmits this to themedia delivery device.

The second feedback section 73 of the wireless base station 10, uponreceiving each RTP packet of the media stream, sends the receptionacknowledge signal (ACK) with the sequence number of the RTP packet backto the media delivery device. In FIG. 9, the ACK 105 is returned inresponse to the reception of the RTP packet 117, the ACK 107 is returnedin response to the reception of the RTP packet 119, and the ACK 109 isreturned in response to the RTP packet 121. Of course, instead oftransmitting the sequence numbers of the packets that were receivednormally back to the media delivery device as explained with respect tothe feedback generator 41′ in FIG. 8, the numbers of the lost packetsmay be sent back, or the arrived/yet unarrived responses may be returnedfor multiple packets grouped together.

Referring to FIG. 8, the second feedback section 73 is provided with themultimedia stream detecting section 37′, the RTP header monitor section39′ and the feedback generator 41′. Functions of each of these sectionsare the same as those of the multimedia stream detecting section 37, theRTP header monitor section 39 and the feedback generator 41, which areincluded in the packet analysis device 15 explained with reference toFIGS. 2 and 3.

The RTCP processing section 113 of the media delivery device 11transfers the packet loss rate and the delay time, which are containedin the RTCP packet 133, to the feedback control section 35. The feedbackcontrol section 35 selects the transmission mode for the media streambased on this information (157).

The above-mentioned three modes are explained below, but explanation isfirst given regarding a method of obtaining parameters that are neededfor the media delivery device 11 to determine the transmission rate andthe transmission mode for the media stream using two types of feedback,Net-Feeds and SP-Feeds, from the wireless base station 10, and feedback(client feedback) from the wireless terminal 19 represented by a line 25in FIG. 8.

First, symbols which will be used are defined as follows:

l₁ Packet loss rate on the wired network 21

R*₁ Maximum transmission rate permitted on the wired network 21

R₁ Transmission rate from the media delivery device 11

R′₁ Reception rate at the wireless base station 10

R₁ ⁽²⁾ Internal rate in wireless base station, after the FEC decoder 72eliminates error correction information for FEC, and before packetdiscard processing is performed at the shaping point 13′

R₁ ⁽³⁾ Internal rate in wireless base station after leaving the shapingpoint 13′

l₂ Packet loss rate on the wireless link (i.e., between wireless device14 and the wireless terminal 19)

R*₂ Maximum transmission rate permitted on the wireless link

R₂ Transmission rate from the wireless base station 10

R′₂ Reception rate at the wireless terminal 19

r Media stream coding rate

Here, the maximum transmission rate R*₂ permitted on the wireless link,R*2, can be determined at the wireless base station 10 by using one ofthe following two methods.

The first method uses ideal maximum transmission rate information in theprocess of setting the wireless link. That is, when setting the wirelesslink, resources assigned to this wireless link, specifically abandwidth, is determined, so that based on this bandwidth the ideal ormaximum possible transmission rate is determined. This value is utilizedas R*₂. In this method, the rate used in the session where this wirelesslink is used, is fixed except in the case where the wireless basestation to be used is replaced, such as when the location of thewireless terminal 19 is changed.

The second method is a method of deriving the maximum transmission rateR*₂ based on the filling rate of the transmission buffer in the wirelessbase station 10 that has the packets that stand ready to be transmittedfrom the wireless base station 10 to the wireless link. That is, if thetransmission buffer's filling rate drops or the buffer keeps empty, thenit is understood that the current transmission rate is below the maximumrate and vice versa, if the filling rate is increasing. The rate R*₂obtained in this way fluctuates depending on the time in many cases.

The maximum transmission rate R*₂ can be determined by, for example, thewireless device 14 in the base station 10, or by a device or the likefor controlling the wireless device 14.

The transmission rates of the media delivery device 11 and the wirelessbase station 10, R₁ and R₂ (i.e., the rates at the transmission ends),are the rates for all packets relating to the media stream, that is, allpackets including not only the main bodies of the data of the mediastream but also the packets for FEC which are added thereto, andretransmitted packets. Because of packet loss that can occur on thewired network or on the wireless link, rates at the respective receptionends, R′₁ and R′₂, drop below the rates at the transmission ends. Inother words, the following formula is established.R′₁≦R₁, R′₂≦R₂

When the packets arrive at the wireless base station 10 from the wirednetwork, first they are given to the FEC decoder 72. If FEC has beenapplied, then the packets are decoded. As a result, the rate R₁ ⁽²⁾after the packets leave the FEC decoder drops below the reception rateR′₁ of the wireless base station.

Here, in the FEC used in this embodiment, the output resulting fromapplying FEC consists of the original data packet and parity packet.Furthermore, the party packet is sent in a separate stream from theoriginal data packet. Regarding this type of FEC, refer to, for example,RFC2733 (An RTP Payload Format or Generic Forward Error Correction). TheFEC decoders (the FEC decoder 72 provided to the wireless base station10 and the FEC decoder (not shown) provided to the wireless terminal 19)use the parity packet sent in the other stream, to reconstruct the lostoriginal data packet.

Before the packet leaves the wireless base station, the FEC generator 18compares the rate R₁ ⁽³⁾ after leaving the shaping point 13′, with themaximum rate R*₂ on the wireless link. If R₁ ⁽³⁾<R*₂, then the FECgenerator 18 uses the surplus bandwidth, R*₂−R₁ ⁽³⁾, for the FEC torestore the packets lost on the wireless link. Eventually, thetransmission rate R₂ from the wireless base station 10 approaches themaximum transmission rate R*₂ permitted on the wireless link.

The shaping point 13′ is provided between the FEC decoder 72 and the FECgenerator 18. When the rate R₁ ⁽²⁾ after eliminating the FEC on thewired network 21, is greater than the wireless link's bandwidth, that iswhen it is greater than the maximum rate R*₂, the packet is discarded.Furthermore, duplicate packets having the same RTP sequence numbers canbe detected and eliminated here as well.

The two types of feedback, Net-Feeds and SP-Feeds, from the wirelessbase station 10 to the media delivery device 11, are sent immediatelybefore the FEC decoder 72 and the FEC generator 18, respectively, viathe first feedback section 71 and the second feedback section 73.

The first feedback section 71, which performs feedback before theshaping point 13′, sends the feedback Net-Feeds to the media deliverydevice 11, to notify the current status on the wired network. Thefeedback Net-Feeds is statistical feedback, and includes informationcollected at a packet window, such as a packet loss rate in a mid-rangetime frame (e.g., several seconds) and the average and variance ofround-trip time. This type of statistical feedback is commonly known. Ifnecessary, refer to, for example, RFC1889 (Rtp: A transport protocol forreal-time application).

The second feedback section 73, which performs feedback after theshaping point 13′, sends the feedback SP-Feeds in the form of a packetreception acknowledge packet (ACK) to the media delivery device 11, sothat the media delivery device 11 can determine which packet is lostbefore the wireless transmission. The feedback SP-Feeds is sent out in ashort period (e.g., within 1 second).

In addition to the feedback Net-Feeds and SP-Feeds, in this embodiment,the wireless terminal 19 also sends out feedback at relatively shortintervals with little delay, to the media delivery device 11.

These types of feedback enable the media delivery device 11 to obtaininformation as listed below.

First, using the feedback Net-Feeds, the media delivery device 11 canknow the maximum transmission rate R*₁ permitted on the wired network.Furthermore, the packet loss rate l₁ on the wired network is explicitlyshown in the feedback Net-Feeds.

The feedback SP-Feeds offers feedback in high resolution at the ends ofthe wired network, which is to say feedback on the packet basis.Therefore, the media delivery device 11 can accurately grasp whichpackets have properly arrived at the wireless base station 10.

Furthermore, using the feedback SP-Feeds and the client feedback fromthe wireless terminal 19, the media delivery device 11 can the obtainpacket loss rate l₂ on the wireless link, and the maximum transmissionrate R*₂ permitted on the wireless link, in the following manner. Thatis, the feedback SP-Feeds is compared against the client feedback foronly the original data packet stream where the parity packet stream forFEC has been eliminated. Thus, the media delivery device 11 can obtainthe packet loss rate after the FEC is applied. Furthermore, from theclient feedback for the parity packet stream, the media delivery device11 can grasp how much FEC has been applied on the wireless link, whichis to say the bandwidth used for the parity packet stream. If the packetloss rate after the FEC is applied, and the bandwidth for parity packetstream (i.e., the parity packet rate) for the FEC, are both known, thenit becomes possible to obtain the “original” loss rate l₂ on thewireless link before the FEC is applied.

In this embodiment, the wireless base station 10 automatically selectsone of the three modes, based on the delay characteristic required ofthe media stream, which is to say the receiving buffer capacity of thewireless terminal 19, and then operates. Hereinafter, explanation isgiven regarding these modes, Mode 1 through Mode 3.

Mode 1

This mode is applied in a case where the wireless terminal 19 (client)is provided with a large receiving buffer, which can allow significantjitter in the packet arrival timing, which is caused by retransmittinglost packets the requisite number of times on the wireless link. Notethat, in third-generation mobile phone communication scheme, thewireless base station 10 may set up the wireless terminal 19 so as toretransmit on the link layer as is commonly known to those skilled inthe art. Its specific construction or operation itself is not directlyrelevant to the present invention. If necessary, refer to H. Holma andA. Toskala, Eds., WCDMA for UMTS: Wireless Access for Third GenerationMobile Communications, Wiley, 2001. Therefore, in order to realize Mode1, a function for retransmitting packets by using third-generationmobile phone link layer can be used. That is, if the wireless basestation 10 sets an error control mode for the wireless terminal 19 tothe retransmitting mode, then, in a manner that is transparent from theapplication layer, the link layer will perform operations as follows.That is, at the link layer, when a lost packet is detected at thewireless terminal 19, the wireless terminal 19 requests the wirelessbase station 10 to retransmit the given packet. When the wireless basestation 10 receives the retransmission request, it compares the numberof times the given packet has been retransmitted to date, with themaximum number of retransmission times which was set when that mode wasset, and then notifies retransmission failure to the wireless terminal19 in a case where the maximum retransmission times has already beenreached. In a case where maximum retransmission times has not beenreached yet, the wireless base station 10 retransmits to the wirelessterminal 19 one of the already-transmitted packets that is temporarilystored and corresponds to the lost packet. The maximum retransmissiontimes of the lost packet is determined based on the packet loss rate l₂on the wireless link, and a target value for improving the packet lossrate by performing retransmission, but an appropriate or standard valuemay be known in advance, or the value may be determined by makingmeasurements as described above. The wireless terminal 19 that hasreceived the retransmitted packet, uses the retransmitted packet torestore the media stream. Therefore, in terms of the application layer,the original error control mode was simply set to the retransmittingmode described above, and thus it looks as though the media stream hasbeen received from the wireless base station 10 without any errors.Accordingly, while there is essentially no packet loss in the wirelesslink from the wireless base station 10 to the wireless terminal 19, agreat deal of variation does occur in the packet transmission delay.

Furthermore, using the feedback Net-Feeds from the wireless base station10, the media delivery device 11 performs congestion control on thewired network.

Furthermore, using the feedback SP-Feeds from the wireless base station10, the retransmission of the lost packet on the wired network isperformed on the application layer. In Mode 1, there are virtually nolost packets on the wireless link, so that the feedback from thewireless base station 10 is substantially the same as the feedback fromthe wireless terminal, but the feedback SP-Feeds arrives at the mediadelivery device 11 considerably faster than the feedback from thewireless terminal.

As compared to FEC, where the error correction code is always added, itis more efficient to perform packet retransmission only when a packet isactually lost. In Mode 1, the delay and jitter which accompany packetretransmission, do not become a problem, so that retransmissions, andnot FEC, are employed on both the wired network and the wireless link.

Mode 2

This mode is used in a case where the size of the receiving buffer inthe wireless terminal 19 is sufficient for the lost packet to beretransmitted the requisite number of times on the wired network, but isnot large enough to allow such retransmission on the wireless link.

In Mode 2, when the packet has been lost on the wired network, in asimilar fashion to Mode 1, the feedback SP-Feeds from the wireless basestation 10 is used by the media delivery device 11 to retransmit thelost packet, and when the packet is lost on the wireless link betweenthe wireless base station 10 and the wireless terminal 19, this issolved by applying FEC.

Normally, the transmission rate on the wireless link is smaller than thewired network 21. Therefore, the typical case is assumed here, that is,R*₁>R*₂. As described above, the media delivery device 11 can obtainR*₁, R*₁, and l₂ from the feedback information. Accordingly, the mediadelivery device 11 can appropriately select a media coding rate (i.e.,the transmission rate for the media stream) r. The bandwidth R*₂−r whichis not used for the media stream on the wireless link, resists the lossrate l₂ on the wireless link as error correction is performed. Thus, anerror correction parity packet stream which is sufficient for restoringvirtually all the lost packets can be transmitted.

On the wired network, the surplus bandwidth R*₂−r is used, to performretransmission on the application layer. Furthermore, since duplicatepackets are discarded at the wireless base station 10, such packets donot occupy bandwidth on the wireless link.

Mode 3

This mode is applied in a case where, because the buffer capacity of thewireless base station 10 is extremely small, any delay cannot beallowed, which is caused by not only the retransmissions on the wirelesslink, but also by retransmitting the lost packet the requisite number oftimes on the wired network.

In Mode 3, the media delivery device 11 applies FEC adaptedappropriately for the error rate and other such error characteristics onthe wired network, and then sends out the media stream toward thewireless base station 10. The parity stream for this FEC is eliminatedby the FEC decoder 72 at the wireless base station 10. The FEC generator18 then applies another FEC adapted appropriately for the errorcharacteristics of the wireless link, and then sends out to the wirelessterminal 19.

Referring back to FIG. 9, the feedback control section 35, based on theRTP packet reception acknowledge signal (ACK) that is sent over from thesecond feedback section 73, determines whether packets in the mediastream are lost (141). When it is detected that a packet is lost, thefeedback control section 35 then determines whether or not thecommunications are being conducted in Mode 1 or Mode 2 (151). When thecommunications are being conducted in Mode 1 or Mode 2, the packetretransmission procedures explained with respect to the blocks 143, 145,147, and 149 in FIG. 6 are then executed (153). When the communicationsare being conducted in Mode 3, if possible and desirable, the errorcorrection code to be used is modified to a more appropriate one (155).

Explanations have been given so far regarding the specific embodimentsof the present invention, but the present invention is not limited tothe above-mentioned embodiments.

1. A media streaming delivery system, comprising: a media deliveryapparatus for transmitting a media stream in packets to a networkaccording to a real time transfer protocol; a relay apparatus connectedto said network for transmitting said media stream to a communicationlink with a large delay; and packet analysis means for monitoring saidpacket arriving at said relay apparatus and transmitting feedbackinformation indicating a status of said network to said media deliveryapparatus; wherein said feedback information is a sequence number of apacket lost from said media stream; wherein said media stream is amoving image including a sequence of I-pictures and P-pictures; andwherein said media delivery apparatus comprises: a storage for storing aplurality of media streams for a same moving image, including at least amedia stream of a first format that includes I-pictures in a firstarrangement and a media stream of a second format that includesI-pictures in a second arrangement which is different from said firstarrangement; and switching means that in response to a determination ofthe lost packet with respect to the destination associated with the lostpacket, selects a media stream from a plurality of said media streams ofat least said first and second formats, the selected media stream beinga media stream in which an I-picture appears earliest after the lostpacket, and for sending the selected media stream to said destination.2. The delivery system according to claim 1, wherein said feedbackinformation is an acknowledge response sent to said relay apparatus eachtime a packet of said media stream arrives.
 3. The delivery systemaccording to claim 1, wherein said communication link with a large delayis a wireless link.
 4. The system according to claim 3, wherein saidmedia delivery apparatus comprises an encoding device for generatingsaid media stream; and wherein said encoding device is arranged togenerate a media stream starting with an I-picture in response todetermination of said packet loss.
 5. A media delivery apparatus usedfor delivering a media stream comprising a sequence of packets, whereinsaid apparatus, in response to feedback relating to packet loss on adelivery path of a media stream, is configured to modify said mediastream so as to reduce an influence of said packet loss; wherein saidmedia stream is a moving image including a sequence of I-pictures andP-pictures, said media delivery apparatus comprising: a storage forstoring a plurality of media streams for a same moving image, saidplurality of media streams including at least a first media stream of afirst format that includes I-pictures in a first arrangement, and asecond media stream of a second format that includes I-pictures in asecond arrangement which is different from said first arrangement; andswitching means for responding to a determination of said packet losswith respect to a destination in which said packet loss has occurred,the switching means configured to select a media stream among saidplurality of media streams, in which an I-picture appears earliest aftersaid packet loss, and for sending the selected media stream to saiddestination.
 6. The media delivery apparatus according to claim 5,wherein said media stream is a video including a sequence of I-picturesand P-pictures; wherein said media delivery apparatus comprises anencoding device for generating said media stream; and wherein saidencoding device is configured to generate a media stream starting withan I-picture in response to determination of said packet loss.
 7. Themedia delivery apparatus according to claim 6, wherein said feedbackrelating to said packet loss is feedback relating to said packet lossobserved at a point where said delivery path switches from a link with asmall delay to a link with a large delay.
 8. The media deliveryapparatus according to claim 7, wherein said link with a small delay isa wired link and said link with a large delay is a wireless link.
 9. Themedia delivery apparatus according to claim 6, wherein said encodingdevice is arranged such that, in response to detection of a packet lossbased on feedback information from said packet analysis apparatus, saidencoding device increases a frequency of I-pictures at least for a mediastream transmitted to a destination for which said loss has occurred.10. The media delivery apparatus according to claim 6, wherein saidencoding device is arranged such that, in response to detection of saidpacket loss based on feedback information from said packet analysisapparatus, said encoding device transmits a media stream starting withan I-picture to the destination for which said packet loss has occurred.11. The media delivery apparatus according to claim 8, wherein saidmedia delivery apparatus performs retransmission of a packet based onfeedback information relating to the packet loss.
 12. A media streamingdelivery method, comprising: transmitting a media stream in packets froma first device to a network according to a real time transfer protocol;relaying said media stream from the network to a communication link witha large delay by using a second device; and monitoring said packets asthey arrive at said second device and transmitting feedback informationindicating the status of said network to said first device; said mediastream includes a video including a sequence of I-pictures andP-pictures; and said first device performing the steps of: storing aplurality of media streams for one video, including at least a firstmedia stream including I-pictures in a first arrangement and a secondmedia stream including I-pictures in a second arrangement which isdifferent from said first arrangement; and responding to a lost packetby (a) selecting a media stream in which a first I-picture after thepicture in said lost packet appears earliest among said plurality ofmedia streams, and (b) switching the media stream to be sent to saidselected media stream, the selecting and switching steps being performedfor the destination for which said loss lost packet occurred.
 13. Thedelivery method according to claim 12, wherein said feedback informationis an acknowledge response sent to said second device each time a packetof said media stream arrives at the second device.
 14. The deliverymethod according to claim 12, wherein said feedback information is asequence number of a packet lost from said media stream.
 15. Thedelivery method according to claim 13, further including modifying saidmedia stream transmitted from the first device based on said feedbackinformation.
 16. The delivery method according to claim 15, wherein saidcommunication link with a large delay is a wireless link.
 17. The methodaccording to claim 12, wherein said media stream includes a video havinga sequence of I-pictures and P-pictures, the method further comprising:generating a media stream starting with an I-picture in response to saidlost packet; and retransmitting the generated media stream based on aacknowledge response received over said wireless link.